Field of the Invention
The present invention relates to an image processing apparatus and method, and more specifically relates to an image processing apparatus and method for detecting motion vectors.
Description of the Related Art
Conventionally, in order to perform image stabilization on a video captured using an imaging apparatus such as a digital still camera or a digital video camera, a moving amount between frame images is detected, and images are aligned by being shifted according to the detected moving amount. Methods for detecting the moving amount between frame images include a method of using information from an external device such as a gyrosensor and a method of estimating the moving amount from captured frame images.
Various proposals have been conventionally made for methods for estimating the moving amount with use of frame images, and a representative one of such methods is motion vector detection by template matching. In template matching, first, one of two captured frame images is set as the primary image, the other one is set as the reference image, and a rectangular area of a predetermined size in the primary image is set as a template block. Then the template block is shifted in the reference image, and a correlation is obtained between primary image pixel values in the template block and reference image pixels values in the template block at each of the shifted positions of the template block. At this time, the position at which the correlation is the highest in the reference image is the destination of the template block, and the direction to the destination and the movement amount that are based on the position of the template block in the primary image are used as the motion vector.
In order to improve the motion vector detection rate, there is a technique in which feature points are extracted, the template block is placed on an extracted feature point, and then template matching is performed between frame images. However, when feature point extraction is performed on an entire image, the distribution of feature points is often uneven, and if a motion vector obtained using uneven feature points is used in image stabilization, the image stabilization becomes focused on an area in which features are concentrated. Image stabilization is performed in order to correct a movement amount in the case where an image has moved by the same amount and in the same direction as a whole due to hand shake, but if the subject itself in the area with a concentration of feature points moves, and that movement is corrected as well, then it is not possible to perform appropriate image stabilization.
In view of this, Japanese Patent Laid-Open No. 2008-192060 discloses a technique in which, in order to distribute feature points evenly in an image, the image is divided into grid blocks, a feature value representing a feature magnitude is calculated for each pixel, and the pixel with the highest feature value in each grid block is extracted as a feature point. FIG. 13 is a diagram based on Japanese Patent Laid-Open No. 2008-192060 for describing a case in which a feature point is obtained for each grid block, template blocks are placed on the obtained feature points, and template matching is performed.
In FIG. 13, 1301 denotes an image, 1302 denotes grid blocks, 1303 denotes feature points in grid blocks, 1304 denotes templates, and 1305 denotes search ranges for templates. Two feature points are shown in this example, and therefore “−1” is added to the reference signs for the elements related to one of the feature points, and “−2” is added to the reference signs for the elements related to the other feature point. For example, in the grid block 1302-1, template matching is performed based on the template 1304-1 that is centered on the feature point 1303-1.
Real-time performance is required in the motion vector detection circuit used in a video camera or the like for image stabilization processing performed on a moving image. For this reason, the circuit is controlled such that, in the grid block 1302-1 for example, template matching processing for the grid block 1302-1 can be started immediately after the completion of image generation for template matching and feature point calculation processing for the grid block 1302-1.
However, as illustrated by the grid block 1302-2 at the lower edge of the image 1301, there are cases where when the feature point 1303-2 is in the bottom portion of the grid block 1302-2, the template 1304-2 and the search range 1305-2 protrude from the bottom of the grid block 1302-2. In such a case, pixel values in an area 1306 below the lower edge grid block 1302-2 are also needed in order to perform template matching. Accordingly, template matching processing cannot be started immediately after the completion of image generation of grid block 1302-2 for template matching and calculation processing for the feature point 1303-2. The start of matching processing becomes delayed until at least image generation for template matching has been completed for the area 1306 outside the grid block 1302-2.
Also, in consideration of the case where a template image and a search range protrude out from a grid block, pixel values need to be prepared in areas outside of grid blocks, and therefore the amount of data written to a memory increases.